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1  Introduction 


The  Defense  Technical  Information  Center  (DTIC)  has  contracted  with  The  Catholic  University 
of  America  to  research  techniques  that  may  help  it  improve  how  it  identifies  user  needs  and 
conducts  requirements  analysis  for  developing  information  retrieval  services  (Statement  of 
Work,  2009).  DTIC  currently  uses  multiple  methods  for  end-users  to  provide  input,  but  such 
input  lacks  the  rigor  of  techniques  used  for  software  design  and  engineering.  DTIC  needs  (1) 
research  into  the  user  needs  and  software  requirements  analysis  techniques  that  are  best  suited 
for  its  information  retrieval  services  and  (2)  application  of  these  techniques  to  document  user 
needs  and  software  requirements.  This  research  will  contribute  to  DTIC’s  understanding  of  the 
potential  future  user  needs  of  its  user  community. 

1. 1  Scope  of  Work 

The  project  was  conducted  in  two  parts.  In  part  1,  we  researched  and  identified  techniques  and 
methods  appropriate  for  DTIC  to  use  for  user  needs  identification  and  related  requirements 
analysis.  We  also  analyzed  DTIC's  current  practices  in  these  areas.  As  the  final  task  of  part  1,  we 
reviewed  with  DTIC  to  jointly  select  specific  techniques  and  deliverables  for  Part  2:  user 
profiles,  personas  and  user  scenarios.  In  Part  2,  we  applied  the  selected  methods  to  research  the 
information  needs,  contexts  of  use,  and  potential  system  requirements  of  DTIC  end-users. 

Part  1  consisted  of  four  tasks: 

•  Task  1:  Review  DTIC’s  current  techniques  and  sources  of  input  from  end-users  regarding 
their  requirements  and  satisfaction  with  their  experience  using  DTIC’s  information 
retrieval  capabilities.  Identify  potential  deficiencies  in  DTIC’s  current  techniques. 

•  Task  2:  Review  needs  elicitation  and  requirements  analysis  techniques  and  identify  those 
that  are  suited  to  DTIC’s  information  retrieval  services  for  scientific  and  technical 
content,  e.g.,  bibliographic  information,  web  sites,  project  descriptions.  Describe  the 
utility,  advantages,  and  disadvantages  of  the  techniques  for  DTIC’s  employment. 

•  Task  3:  Recommend  the  priority  sequence  for  DTIC’s  employment  of  the  techniques 
considering  factors  such  as  DTIC’s  end-user  characteristics,  applicability  to  DTIC’s 
services,  criticality  of  DTIC’s  information  deficiencies  and  cost/effectiveness  trade-offs. 

•  Task  4:  Present  written  findings  (this  report)  to  DTIC.  Review  findings  with  DTIC  to 
jointly  determine  specific  techniques  to  apply,  and  deliverables  to  produce,  for  Part  2. 

Part  2  consisted  of  2  tasks: 

•  Task  1:  Apply  two  of  the  recommended  techniques  to  produce  a  set  of  deliverables 
documenting  user  needs  and  potential  system  requirements.  Techniques  and  deliverables 
provided  will  be  based  on  DTIC-specific  characteristics,  e.g.,  DTIC  services,  DTIC 
processes,  user  characteristics,  system  migration  plans,  content  of  DTIC’s  databases. 
Examples  of  deliverables  include  user  scenarios,  questionnaires,  screen  shots,  process 
models,  and  use  cases. 
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•  Task  2:  Present  customized  techniques  to  the  DTIC  point  of  contact  no  later  than  four 
months  from  the  date  of  contract  execution  [note:  contract  was  extended].  Provide 
electronic  products  in  a  form  that  is  usable  by  DTIC  without  the  purchase  of  new 
technology. 

1.2  Principal  Investigator  Qualifications 

Dr.  Bill  Kules  is  an  Assistant  Professor  in  the  School  of  Library  and  Information  Science  (SLIS) 
at  The  Catholic  University  of  America.  His  research  interests  include  the  design  and  evaluation 
of  rich  interfaces  to  support  complex  information  seeking  and  retrieval  tasks  like  exploratory 
search.  Dr.  Kules  is  an  organizer  of  the  workshops  on  Human-Computer  Interaction  and 
Information  Retrieval  (HCIR).  He  co-chaired  HCIR  2009,  was  Program  Chair  of  HCIR  2008, 
and  is  on  the  steering  committee  of  HCIR  2010.  He  was  an  organizer  of  the  2005  Workshop  on 
Exploratory  Search  Interfaces  and  a  guest  editor  for  the  April  2006  CACM  Special  Section  on 
Supporting  Exploratory  Search.  In  conjunction  with  his  teaching,  he  is  investigating  how  the 
library  and  information  science  field  teaches  students  about  information  technology.  Before 
joining  SLIS  Dr.  Kules  spent  20  years  designing  and  implementing  information  systems  for  a 
variety  of  applications,  include  wireless  telephony,  customer  service  and  banking.  He  earned  his 
Ph.D.  in  Computer  Science  at  the  University  of  Maryland. 

1.3  Methods 

The  research  employed  multiple  methods  to  gather  data: 

•  Documentation  review  -  We  reviewed  documentation  available  on  DTIC's  web  site  and 
documents  made  available  to  us.  All  material  was  unclassified  and  unrestricted.  See 
appendix  A  for  a  list  of  documents  reviewed. 

•  One-on-one  and  small  group  interviews  -  For  part  1,  we  interviewed  17  staff  members 
from  DTIC-A,  B  and  E,  either  in  person  or  via  telephone.  These  people  are  involved  in 
needs  assessment,  requirements  analysis,  design  specification,  and  related  activities.  For 
part  2,  we  conducted  in-depth,  semi-structured  interviews  with  6  DTIC  end-users. 

•  Attended  executive  briefing  by  current  search  engine  vendor  (Microsoft  FAST). 

•  Literature  review  -  We  conducted  a  literature  search  on  military  information  seeking  and 
related  user  needs  identification.  We  also  reviewed  selected  user  needs  identification 
techniques  in  the  areas  of  library  and  information  science  (LIS),  human-computer 
interaction  (HCI),  user  interface  design,  and  systems  analysis. 

In  part  1,  data  gathered  from  the  interviews  and  document  review  was  used  to  document  DTIC's 
current  operational  process  for  needs  identification,  requirements  analysis  and  generating  design 
specifications.  This  information  was  used  to  preliminarily  critique  the  current  process.  This 
analysis  was  narrowly  focused  on  aspects  related  to  user  needs,  so  the  documentation  should  not 
be  considered  comprehensive  of  the  broader  requirements  analysis  or  design  process.  The 
literature  review  was  used  to  extend  our  understanding  of  user  needs  identification  to  DTIC's 
domain:  defense-related  information  seeking. 
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In  part  2,  we  conducted  semi- structured  interviews  with  DTIC  end-users.  We  contacted  17  DTIC 
end-users,  identified  through  their  use  of  DoDTechipedia  or  through  contacts  in  the  COCOMs. 
Of  these,  we  interviewed  4  DoDTechipedia  users  and  2  COCOM  contacts.  The  interview 
protocol  was  approved  as  exempt  by  the  Catholic  University  Institutional  Review  Board.  Each 
interviewee  agreed  in  advance  to  be  recorded.  The  interviews,  which  lasted  30-45  minutes  each, 
were  recorded,  transcribed,  and  then  analyzed  to  characterize  these  users  and  their  information 
needs  through  the  creation  of  personas  and  user  scenarios,  which  are  descriptive  narratives  often 
used  in  systems  design  and  development.  See  Appendix  B  for  a  detailed  description  of  the 
activities  and  Appendix  C  for  the  questions  used  for  the  interviews. 

2  Assessment  of  Current  DTIC  Practices  for  User  Needs 
Identification 

In  Task  1  we  first  sought  to  understand  the  current  operational  process  for  needs  identification, 
requirements  analysis  and  generating  design  specifications.  We  used  this  information  to  suggest 
initial  answers  to  the  following  questions: 

•  Does  the  process  consistently  ensure  that  end-user  needs  are  considered  and  addressed  in 
design  specifications? 

•  Are  there  opportunities  to  improve  this  process? 

•  Are  there  potential  sources  of  input  not  being  gathered? 

DTIC  serves  a  variety  of  end-users,  from  highly-trained  information  professionals, 
intermediaries  and  researchers  (experts)  to  searchers  without  any  search  training  (informal 
searchers).  These  are  all  part  of  DTIC's  primary  audience,  the  defense  community.  Traditionally, 
DTIC  has  extensive  contact  with  the  experts,  and  the  needs  of  that  group  are  fairly  well 
understood  and  supported.  There  is  a  growing  recognition  of  the  need  to  support  informal 
searchers.  The  Strategic  Plan  explicitly  acknowledges  non -traditional  DTIC  clients.  There  is  an 
effort,  for  example,  to  make  connections  to  the  combatant  commands.  However,  this  recognition 
appears  to  be  inculcated  to  different  degrees  within  DTIC.  My  conversations  suggest  that  much 
of  DTIC's  staff  contact  is  with  experts  rather  than  informal  searchers. 

The  needs  of  informal  searchers  are  not  as  well  understood  as  the  needs  of  the  professionals.  It  is 
not  clear  who  comprises  the  informal  searchers.  DTIC’s  User  Services  Directorate  indicates  that, 
"We  usually  advertise  that  we  target  the  science,  technology,  research  and  development, 
acquisition,  and  warfighter  communities.  Our  Web  page  refers  to  serving  the  warfighter, 
researchers,  scientists,  engineers,  laboratories,  and  universities"  (private  communication). 
However,  I  have  found  no  document  yet  that  articulates  the  types  of  users  DTIC  seeks  to  serve 
within  each  of  these  large  and  diverse  communities.  This  is  an  important  first  step  in  a  deeper 
understanding  of  the  needs  of  users. 

During  the  FAST  Executive  Briefing,  the  discussion  I  heard  suggested  that  the  1 1  Big  Ideas 
reflect  individual  judgments  about  a  small  set  of  technological  capabilities.  For  example, 
collaborative  search  in  virtual  worlds  is  one  of  the  ideas,  but  non  VR-based  collaborative  search 
interfaces  were  not  mentioned.  I  did  not  hear  reference  to  any  user  needs  assessment  that  was 
guiding  the  decision-making.  The  Customer  Survey  might  be  a  useful  piece  of  data,  but  it 
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reflects  a  biased  sample  of  DTIC  users  (mostly  experts). 

DTIC  has  multiple  developmental  activities,  with  a  disparate  variety  of  sponsors,  clients,  and 
end-users.  Some  projects  are  externally  mandated,  with  time  constraints  and  functional 
requirements  explicitly  defined,  over  which  DTIC  has  limited  control.  For  example, 
implementation  timelines  for  both  DoDTechipedia  and  DTIC  Online  Public  appear  to  have  been 
driven  in  part  by  external  deadlines.  That  appears  to  have  limited  DTIC's  ability  to  fully  gather 
user  requirements,  prototype  and  refine  the  designs  for  these  projects  prior  to  implementation 
and  launch. 

It  isn't  clear  that  DTIC  has  a  standard  development  process  that  is  consistently  applied.  One 
interviewee  indicated  that  they  use  the  Scrum  framework  for  at  least  some  projects.  In  any  case,  I 
found  little  awareness  of  such  a  process  during  the  interviews.  This  is  important  because  having 
a  variety  of  development  processes  makes  it  harder  to  ensure  that  user  requirements  analysis  (as 
well  as  other  project  activities)  is  done  consistently  and  thoroughly.  In  particular,  this  can  lead  to 
frustration  within  the  organization  because  people  with  valuable  knowledge  of  end-user  needs  do 
not  have  an  appropriate  way  to  contribute  to  the  user  requirements  gathering  process  or  related 
parts  of  a  project  (e.g.  prototype  reviews,  informal  usability  testing).  The  Customer  Interface 
Task  Force  appears  to  be  a  good  step  in  building  a  cross-functional  team  to  address  this  issue.  It 
does  not  appear  to  represent  the  full  spectrum  of  DTIC  users. 

3  Review  of  Selected  User  Needs  Identification  and 
Requirements  Analysis  Techniques 

The  initial  intent  of  task  2  was  to  review  techniques  for  identifying  user  needs  and  related  system 
requirements.  As  I  pursued  this  task,  I  realized  that  there  is  a  precursor  need:  identifying  who  the 
users  are.  This  review  focuses  on  techniques  and  activities  related  to  both  aspects,  after  a  brief 
review  of  our  limited  findings  on  military  information  needs.  The  section  closes  with  a 
discussion  of  several  general  information  gathering  techniques  that  may  be  applicable  to  this 
project. 

3. 1  Military  Information  Needs  and  Information  Seeking 

The  review  of  literature  covering  military  information  needs  and  information  seeking  suggests 
that  there  is  much  attention  to  battlefield  information  needs  (Dolgoff ,  1997;  Kahan  et  al.,  1989; 
Kuylenstierna  et  al.,  2004;  Sinclair,  1997;  Thoms,  2003)  and  information  sharing  (Dearstyne, 
2005;  Kaiser,  2003;  Feinberg,  2004;  McWilliams,  1991;  Stowsky,  2004).  There  is  some 
discussion  of  the  history  and  role  of  DTIC  (Molholm,  1995),  and  the  federal  cataloging  system 
(Hayes,  1992).  One  recent  study  has  examined  information  seeking  related  to  government 
information  (Burroughs,  2009),  although  this  is  not  specific  to  military  information  needs. 

In  summary,  there  appears  to  be  little  attention  in  the  research  and  professional  literature  related 
to  science  and  technology  information  needs  of  the  military  community,  and  no  attention  to 
trends  in  this  area. 
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3.2  Identifying  and  Understanding  Users 


Stakeholders  are  groups  or  individuals  that  have  an  interest  in  DTIC's  services  (and  are  part  of  its 
target  communities).  They  may  be  identified  by  group  membership,  e.g.,  formal  organizations 
like  the  Department  of  Defense,  or  categories  of  users,  like  defense  contractors  or  military 
service  schools.  Users  may  be  identified  via  shared  characteristics  or  interests,  such  as  an  interest 
in  chemical/biological  defense.  They  may  also  be  identified  by  specialized  roles  (e.g.,  chemical 
engineer,  teaching  faculty,  logistics  officer,  manager,  program  officer).  It  is  particularly 
important  to  identify  "invisible"  users  who  rarely  interact  with  staff  (Westbrook,  45). 

Segmentation  is  the  process  of  classifying  customers  into  groups  with  different  needs, 
characteristics  or  behavior  (Walters,  33-35).  Identifying  groups  with  specific  patterns  of 
information  needs  will  allow  DTIC  to  focus  resources  where  they  can  be  used  most  effectively 
and  to  meet  the  needs  of  previously  underserved  or  newly  developed  groups.  Broadly  targeted 
services,  such  as  online-searching  capabilities  through  DTIC's  web  site,  can  be  further  developed 
to  facilitate  information  retrieval  for  specific  segments  of  information  seekers. 

Classifying  groups  with  interests  in  DTIC's  services  can  be  achieved  by  identifying  variables 
such  as  the  organization  that  they're  associated  with,  their  particular  areas  of  interest,  the  level  of 
services  that  they  need,  or  their  role  or  level  of  influence  within  an  organization.  Programs  and 
resources  can  be  effectively  developed  by  examining  what  services  a  particular  group  has 
available  to  it,  the  manner  in  which  the  services  reach  it,  and  the  ends  to  which  these  services  are 
applied. 

Demand  refers  to  the  total  number  of  potential  customers  in  a  particular  segment.  As  groups  are 
more  narrowly  segmented,  it  becomes  easier  to  comprehend  the  actual  demand  for  a  product  or 
service.  Factors  such  as  intensity  of  demand  and  timing  or  rhythm  of  demand  can  assist  in 
identifying  key  needs  for  individual  programs.  Forecasting  demand,  especially  in  cases  where  a 
particular  segment  experiences  fluctuations  in  volume  or  types  of  needs,  can  assist  in  making 
sure  that  a  segment's  needs  are  met  and  that  DTIC's  resources  are  prepared. 

Information  about  users  and  user  groups  can  be  gathered  through  primary  information  (both 
quantitative  and  qualitative)  such  as  questionnaires  and  focus  groups  and  secondary  information 
such  as  research  reports  and  case  studies.  Frequently,  direct  information  from  users  themselves 
can  be  supplemented  and  expanded  upon  through  analytical  materials  to  help  an  organization 
decide  what  direction  to  take  with  a  particular  resource  or  service. 

Exposure  to  end-users  can  help  in  determining  ways  to  improve  a  system's  usability.  "User  roles 
may  be  similar  in  style  of  interaction,  in  expectation,  or  in  terms  of  any  of  a  variety  of  common 
characteristics  or  shared  features."  (Constantine  &  Lockwood,  71-96)  "What  distinguishes  how 
a  given  user  would  or  could  use  the  system/characterizes  their  relationship  to  the  system?" 
Different  types  of  user  roles  and  their  dynamic  relationships  to  each  other  may  also  affect  the 
usability  of  a  system  as  a  whole. 

User  profiles  may  be  used  to  capture  demographic  and  other  attributes  of  stakeholders  or  other 
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targeted  populations  of  users.  Age,  education,  training,  experience,  job  functions,  and  common 
tasks  may  be  included  in  a  user  profile. 

Personas  may  also  be  used  as  a  way  to  describe  system  users  throughout  the  design  process. 
Personas  are  fictional  characters,  not  real  people,  but  they  are  based  on  the  behaviors  and 
motivations  of  real  people  that  have  been  observed  (Cooper  et  al,  75-86).  Personas  often,  but  not 
always,  include  a  name,  a  picture,  and  details  such  as  age,  education,  and  experience.  They  can 
include  job  titles  as  well  as  details  about  the  environment  they  work  in.  Personas  provide  a 
concrete  way  of  thinking  and  communicating  about  how  users  behave,  how  they  think,  what  they 
wish  to  accomplish,  and  why.  They  make  abstract  user  groups  more  salient,  which  can  help 
designers  and  developers  better  understand  and  support  end-user  needs.  Creating  a  persona  that 
is  broad  in  functionality  in  order  to  accommodate  the  most  people  will  not  necessarily  be 
effective.  Instead,  creating  a  range  of  personas  representing  specific  types  of  individuals  with 
specific  needs  will  enable  a  more  realistic  view  of  how  a  user  might  interact  with  a  system. 

The  definition  of  user  may  be  expanded  to  include  those  who  manage  direct  users,  those  who 
receive  products  from  the  system,  those  who  test  and  repair  the  system,  those  who  make  the 
purchasing  decision,  and  those  who  use  competitive  products  (Sharp  et  al,  430-431).  Primary 
users  are  those  likely  to  be  frequent  hands-on  users  of  the  system;  secondary  users  are  occasional 
users  or  those  who  use  the  system  through  an  intermediary;  and  tertiary  users  are  those  affected 
by  the  introduction  of  the  system  or  who  will  influence  its  purchase.  These  types  of  users  may 
be  found  within  a  particular  segmented  group  or  constitute  a  group  of  their  own.  They  may  be 
overlooked  in  systems  design  efforts  because  they  are  not  directly  represented.  In  DTIC’s  case, 
these  users  could  be  anyone  who  is  acquires  information  from  DTIC,  even  indirectly,  e.g. 
through  an  intermediary. 

3.3  User  Needs  Assessment 

Once  the  stakeholders  have  been  identified,  needs  assessment  and  requirements  identification 
can  proceed.  In  practice,  the  assessment  process  may  identify  additional  users,  particularly 
tertiary  users,  who  are  not  direct  users  of  the  system.  In  this  section  I  survey  techniques  used  to 
identify  needs  and  requirements. 

User-centered  design  is  an  accepted  practice  in  software  development.  A  user-centered  approach 
means  that  the  real  users  and  their  goals  should  be  the  driving  force  behind  development  of  a 
system  or  service  (Sharp  et  al,  2007,  425-426).  In  taking  a  user-centered  approach  to 
implementing  a  system  user's  behavior  and  context  of  use  are  studied  and  the  system  is  designed 
to  support  them,  rather  than  having  the  users  adapt  their  behavior  to  the  system  and  its 
capabilities.  Iterative  design  (cycles  of  designing,  testing,  measuring,  and  redesigning)  allows 
system  development  to  be  refined  based  on  feedback.  This  allows  the  final  product  to  evolve  in  a 
manner  that  is  tailored  to  user  needs,  rather  than  simply  appearing  whole  and  unalterable. 

Involving  users  throughout  the  development  process  will  help  ensure  that  a  system  will  take  user 
needs  and  activities  into  account,  both  for  broad  groups  of  users  and  specific  segments  of  the 
user  population.  "Expectation  management  is  the  process  of  making  sure  that  the  users'  views 
and  expectations  of  the  new  product  are  realistic.”  (Sharp  et  al,  418-420)  Users  can  be  involved 
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in  a  variety  of  ways,  from  being  part  of  the  design  team  to  participating  in  workshops  and 
evaluation  sessions,  or  just  being  kept  informed  through  newsletters  or  announcements.  User 
involvement  can  occasionally  lead  to  problems  with  unfulfilled  expectations  and  interruptions  in 
the  development  process,  but  appropriately  structured  interactions  with  users  can  help  prevent 
this.  DTIC  has  a  strong  and  involved  user  community,  which  can  be  leveraged.  Because  this 
community  reflects  a  portion  of  the  overall  audience,  additional  outreach  would  be  needed  to 
users  not  already  represented. 

Asking  users  to  identify  their  needs  can  be  tricky  because  users  are  not  always  aware  of  what's 
possible.  "Instead,  we  have  to  approach  it  by  understanding  the  characteristics  and  capabilities  of 
users,  what  they  are  trying  to  achieve,  how  they  achieve  it  currently,  and  whether  they  would 
achieve  their  goals  more  effectively  and  have  a  more  enjoyable  experience  if  they  were 
supported  differently.  Introducing  a  new  product  to  the  user  population  requires  a  culture  change, 
which  is  not  always  easy  to  create.  (Sharp  et  al.,  2007,  432-433)  A  system  designed  to  reference 
back  to  users’  previous  habits  and  behaviors  can  help  ease  the  transition.  Thus  some  system 
requirements  may  be  transitional. 

System  design  involves  multiple  forms  of  data  gathering,  including:  interviews,  focus  groups, 
questionnaires,  direct  observation,  indirect  observation,  studying  documentation,  researching 
similar  products.  Data  on  user  needs  must  cover  a  wide  spectrum  of  issues,  so  it's  preferable  to 
take  a  variety  of  approaches  to  expand,  clarify,  and  confirm  the  data  that  is  out  there.  "We  need 
to  find  out  about  the  tasks  that  users  currently  perform  and  their  associated  goals,  the  context  in 
which  the  tasks  are  performed,  and  the  rationale  for  the  current  situation."  (Sharp  et  al.,  2007, 
489-500).  Interviews  can  be  effective  in  the  early  stages  of  development  for  getting  people  to 
explore  issues  and  describe  scenarios  that  can  be  used  for  identifying  problems  with  the  current 
system  and  suggesting  alternatives.  "Focus  groups  are  good  at  gaining  a  consensus  view  and 
highlighting  areas  of  conflict  and  disagreement  during  the  requirements  activity."  Focus  groups 
can  help  in  gaining  an  understanding  the  particular  needs,  attitudes,  and  problems  of  targeted 
segments  of  user  population  (Walters,  2004,  55-60).  Focus  group  participants  should  be  chosen 
for  their  similarities  to  one  another  with  regard  to  the  issues  that  the  focus  group  will  be 
discussing,  rather  than  as  representatives  of  a  diverse  user  population.  Personal  interviews  may 
allow  a  researcher  to  dig  deeper  into  a  particular  issue  and  provide  an  opportunity  for  researchers 
to  gain  more  complex  answers  to  their  questions.  Interviews  and  focus  groups  are  flexible  and 
relatively  easy  to  organize.  They  provide  the  opportunity  to  ask  and  follow  up  on  detailed 
questions.  Their  small  sample  size  means  that  they  provide  anecdotal  rather  than  comprehensive 
data. 

Questionnaires  can  be  effectively  used  to  gain  perspective  on  specific  issues  throughout  a  broad 
spectrum  of  users  and  initiate  feedback/suggestions.  In  the  government  environment,  fielding  a 
questionnaire  requires  review  and  approval  by  the  Office  of  Personnel  and  Management,  which 
adds  time  and  complexity.  In  addition,  questionnaires  must  be  publicized  or  otherwise  presented 
to  that  target  community  and  incentives  may  be  needed  for  completion.  Nevertheless, 
questionnaires  are  efficient  tools  for  reaching  a  large  population  of  users  to  address  specific, 
quantifiable  questions. 
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Direct  observation  of  users  in  their  regular  settings  can  assist  in  understanding  the  context  under 
which  a  user  engages  in  activity  with  a  system  and  the  types  of  tasks  that  are  used  to  fulfill  their 
needs.  Indirect  observation,  such  as  diaries  and  interaction  logging,  can  be  used  to  gain  similar 
information  without  intruding  on  the  user  environment.  Contextual  inquiry  is  a  specialized  form 
of  direct  observation,  wherein  the  designer  adopts  a  role  similar  to  an  apprentice  to  the  user. 
During  a  contextual  interview  the  designer  and  user  work  collaboratively  to  observe,  discuss  and 
interpret  the  user's  tasks  and  activity  in  a  system.  Any  of  these  techniques  require  on-site  access 
to  users  and  the  willingness  of  the  informant  to  be  observed  and  possibly  interrupted  with 
questions.  Security  constraints  and  the  disparate  location  of  DTIC  users  reduce  the  viability  of 
this  approach  for  DTIC. 

Documentation  such  as  manuals  and  help  logs  can  provide  useful  information  on  the  steps 
involved  with  specific  tasks  and  where  users  are  currently  experiencing  problems.  Researching 
similar  products  and  performing  competitive  evaluation  can  generate  alternative  designs  and 
prompt  ideas  for  new  system  requirements.  The  documentation  of  interest  in  this  project  is 
unlikely  to  be  classified,  although  it  may  be  intended  for  limited  distribution. 

User  scenarios  provide  narrative  descriptions  of  user  tasks  that  allow  for  exploration  of  user 
needs  and  system  requirements  (Sharp  et  al.,  2007,  505-518).  They  can  be  developed  through  a 
number  of  techniques,  including  focus  groups  and  expert  consultation.  User  scenarios  use  the 
vocabulary  and  phrasing  of  users  and  provide  a  natural  way  for  people  to  explain  how  they're 
doing  things  and  highlight  potential  design  ideas  for  a  new  system.  Scenarios  detail  how  their 
characteristic  elements  interact  with  each  other  (Carroll,  200,  46).  For  example,  the  setting, 
actors,  events,  and  goals  can  all  interact  with  one  another  and  change  the  nature  of  other 
elements  or  the  task  at  hand  altogether.  They  provide  an  explicit  description  of  how  a  task  is 
handled  and  the  variables  that  influenced  the  decisions  made  by  the  user,  as  well  as  the  ultimate 
outcome.  This  is  particularly  useful  when  designers  and  developers  are  not  familiar  with  the  end- 
users,  as  is  the  case  with  DTIC’s  non-traditional  end-users.  User  scenarios  can  provide  insight 
into  why  users  do  or  do  not  use  a  system;  environmental  or  organizational  constraints  that  affect 
their  use  of  a  system;  and  what  capabilities  or  services  they  might  find  valuable  in  satisfying 
information  needs  and  achieving  business  objectives. 

Another  technique,  use  cases,  can  provide  similar  information,  but  it  differs  from  user  scenarios 
in  that  the  focus  is  on  describing  the  user-system  interaction  through  a  dialog  approach.  Use 
cases  typically  include  multiple  possible  outcomes,  whereas  user  scenarios  focus  on  one  specific 
example.  However,  in  both  techniques  the  focus  is  still  "on  the  user's  perspective,  not  the 
system's." 

Hierarchical  Task  Analysis  (HTA)  is  used  to  investigate  existing  situations  by  breaking  a  task 
into  subtasks,  and  then  further  into  subtasks  as  necessary.  This  provides  a  high  level  of  detail  for 
examining  how  a  complex  system  works  as  a  whole.  It  is  appropriate  when  specific  tasks  require 
a  detailed  understanding  or  when  there  are  a  few  ways  of  achieving  the  task  that  are  to  be 
optimized.  In  the  context  of  information  seeking,  HTA  is  more  appropriate  for  detailed  system 
design  rather  than  the  long  range,  high-level  focus  of  this  project. 

"The  overall  purpose  of  requirements  analysis  is  to  determine  user,  stakeholder,  and 


organizational  needs."  (Stair  &  Reynolds,  2008,  511)  Requirements  analysis  specifically  asks  the 
involved  parties  about  their  satisfaction  with  the  current  system  and  what  improvements  could  be 
made  to  the  system  in  order  for  it  to  function  more  effectively.  It  can  help  determine  Critical 
Success  Factors  (CSFs),  aspects  that  are  vital  to  the  success  of  an  organization.  CSFs  can  be 
materials  or  information  that  are  retrieved  or  acquired  through  the  system  and  allow  other 
processes  to  occur  or  continue.  As  with  HTA,  CSFs  are  more  appropriate  for  detailed  analysis 
and  design  tasks. 

3.4  Other  Techniques 

The  techniques  detailed  above  are  well  accepted  for  defining  current  requirements  for  a  system. 
That  is,  they  are  useful  for  eliciting  current  needs.  Since  the  focus  of  this  project  is  to  identify 
future  needs,  DTIC  should  also  consider  techniques  that  are  more  “forward  looking.”  There  are 
several  ways  to  do  this. 

Workshops  (essentially  focus  groups  of  experts)  are  used  in  the  research  community  to 
investigate  specific  aspects  of  a  topic.  They  provide  a  forum  for  focused  discussion  and 
consensus-building.  One  to  two  day  workshops  can  be  structured  to  yield  clarify  a  question, 
generate  recommendations,  or  critique  an  approach.  In  the  context  of  this  project,  a  workshop 
could  be  used  to  identify  important  trends  in  information  needs  and  behaviors  and  develop  a  list 
of  anticipated  information  needs.  Unlike  other  focus  groups,  workshops  require  extensive 
planning.  Experts  need  to  be  recruited,  their  schedules  need  to  be  coordinated  well  in  advance, 
and  they  need  to  be  compensated  for  their  time  and  travel.  The  workshop  needs  to  be  carefully 
structured  and  facilitated.  Participants  may  need  to  be  asked  to  do  pre-work.  Experts  bring  their 
own  biases,  so  workshop  organizers  need  to  be  aware  of  them  and  recruit  a  balanced  group. 
Nevertheless,  if  the  expense  and  scheduling  constraints  are  acceptable,  workshops  can  be  very 
effective  tools. 

Case  studies  provide  in-depth  knowledge  based  on  existing,  real-world  systems  organizations  or 
systems.  Case  studies  can  be  conducted  relatively  inexpensively,  because  typically  one  or  two 
people  conduct  the  research  and  analysis.  Typically  one  organization  or  system  is  selected  for  the 
study.  A  structured  set  of  information  is  compiled  and  analyzed  to  answer  specific  questions. 
Although  case  studies  may  involve  other  techniques,  such  as  interviews  or  focus  groups,  they 
rely  heavily  on  existing  documentation.  Thus,  they  are  limited  by  the  availability  of  informants 
and  documentation.  Since  the  focus  is  on  the  present  or  past,  they  are  not  inherently  forward- 
looking,  although  the  findings  can  be  used  to  suggest  future  trends. 

Environmental  scans  can  provide  early  detection  of  important  trends.  Scans  may  be  a  one-time  or 
ongoing  activity.  They  depend  on  having  a  broad  set  of  resources  that  can  be  effectively 
monitored  and  analyzed.  Scans  are  limited  by  the  quality  and  timeliness  of  the  resources.  In  the 
context  of  this  project,  a  scan  would  need  to  include  research  journals  related  to  information 
needs  and  behaviors,  especially  of  the  “Millennial”  and  succeeding  generations.  These  would 
include  journals,  conferences  and  workshop  in  the  LIS  field,  as  well  as  education,  sociology, 
human-computer  interaction  (HCI),  psychology,  and  other  fields. 
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4  User  Profiles,  Personas  and  Scenarios 

Following  discussion  of  the  part  1  findings,  we  agreed  to  apply  three  techniques,  producing  the 
following  three  types  of  user  and  task  descriptions: 

•  User  profiles 

•  Personas 

•  Scenarios 

4. 1  User  profiles 

For  this  study,  we  collected  a  small  set  of  attributes  for  the  user  profiles  (see  Table  1): 

•  Position 

•  Education  level 

•  Gender 

•  Age  (range) 

•  Work  experience  -  both  in  the  field  and  at  the  current  job 

•  Web  search  frequency  and  experience 
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Table  1.  User  profiles  of  the  six  end-users  interviewed  in  part  2  of  the  project. 


Position 

Educ. 

Level 

Gender 

Age 

Years  in 
field 

Years  at 
this  job 

Web  search 
frequency 

Web 

searching 

skill 

Scientist,  USNR 
officer,  technical 
consultant,  DDR&E 

PhD 

M 

50-59 

22 

6 

Daily 

3.5 

Analyst,  contractor, 
ASDNII/DoD  CIO 

n/r 

F 

n/r 

n/r  n/r 

Daily 

4.5 

Civilian  analyst, 
NAVSEA 

Some 

college 

M 

50-59 

35 

6 

Daily 

4.5 

Systems  analyst, 
contractor,  MDA 

MS, 

MBA 

M 

50-59 

30  n/r 

Daily 

4.5 

Systems  analyst, 
contractor,  DDR&E 

BS 

F 

30-39 

10 

0.5 

Daily 

4 

Analyst,  AF  Space 

BS,  MS 

M 

40-49 

23  n/r 

Daily 

4 

Command 


4.2  Personas  and  Scenarios 

From  the  six  interview  transcripts,  I  created  four  personas  and  seven  scenarios  of  information 
needs.  Specific  personal  and  task  details  have  been  merged  and  altered  to  avoid  directly 
identifying  individuals  while  remaining  true  to  the  information  need,  sources  consulted  and 
search  strategies  and  tactics. 

Charles  -  Industrial  process  improvement 

Charles  is  responsible  for  process  improvement  in  naval  shipyards.  He  regularly  looks  for 
specific  technical  ways  to  improve  operations.  He  also  looks  for  resources  to  help  large 
organizations  change  technical  cultures  and  improve  communication  in  distributed  organizations. 
He  uses  DoDTechipedia  extensively  to  share  new  ideas  and  techniques  with  tradesmen  around 
the  country.  He  does  not  currently  use  other  DTIC  services  much  because  he  finds  them 
“cumbersome”  even  though  they  may  contain  useful  information. 

Specific  Need  -  Charles  is  researching  an  application  of  the  Metal  Inert  Gas  (MIG)  welding 
process  applicable  to  working  with  assault  amphibious  vehicles  (AAV’s).  He  does  not  know 
where  to  begin  to  look  for  authoritative  information  sources  on  this  application,  so  he  starts  with 
Google.  Google  returns  a  result  set  which  includes  the  Electric  Welding  Institute.  Although  he 
didn’t  expect  to  find  a  result  from  this  organization,  he  knows  of  it  and  is  already  a  member,  so 
he  logs  in  and  from  there  locates  specific,  known  information  and  also  “joins  the  conversation” 
to  ask  how  to  apply  this  process  in  the  Navy  yards.  He  spends  a  total  of  about  half  an  hour  on 
this  search. 

Exploratory  Need  -  Over  the  next  several  weeks,  in  small  chunks,  Charles  continues  searching  on 
this  application  of  MIG  welding.  His  goal  is  to  better  understand  the  current  limitations  of  the 
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process  and  look  for  improvements.  The  Electric  Welding  Institute  is  already  a  marked 
“favorite”  of  his.  Marking  a  web  resource  as  a  favorite  enables  him  to  easily  recall  known 
resources  that  he  has  previously  found  and  vetted.  As  part  of  his  ongoing  research,  he  looks  for 
availability  and  configuration  information  for  the  machines  and  tools  needed.  With  each  search, 
he  takes  the  inquiry  further  in  anticipation  of  addressing  potential  concerns  that  the  welders 
might  have.  This  yields  no  additional  technical  information  but  puts  him  in  touch  with  more 
subject  matter  experts.  He  ultimately  decides  to  continue  monitoring  the  technique  so  that  it  can 
eventually  be  considered  for  field  applications. 

Roberta  -  Technical  analyst 

Roberta  works  for  the  Air  Force  Space  Command.  She  is  involved  in  a  variety  of  tasks  related  to 
information  systems  -  some  strategic  planning,  contract  administration,  architecture  and 
engineering.  She  searches  the  web  daily.  She  uses  DTIC  services  less  than  once  a  month.  She 
hasn’t  yet  found  relevant  content  on  DoDTechipedia. 

Exploratory  Need  -  Roberta  hears  about  the  IT  Infrastructure  Library  from  a  colleague  and  wants 
to  learn  more  about  it  to  improve  the  quality  of  her  document  reviews.  She  starts  with  a  Google 
search  and  finds  the  official  web  site,  but  she  wants  information  that  will  help  her  directly  apply 
this  within  MDA,  so  she  continues  searching,  using  Google  and  the  AF  Portal.  Her  searches 
frequently  return  results  that  appear  promising,  but  turn  out  to  be  a  vendor  site  or  some  other 
resource  which  requires  subscription  for  information.  She  looks  for  the  best  sources  that  are  not 
charging  for  the  information,  because  expending  funds  every  time  for  sources  she  encounters 
presents  budget,  time  and  aggravation  factors.  After  spending  about  4  hours  over  2-3  weeks,  she 
gets  several  book  recommendations  from  a  contractor  she  trusts,  and  gets  authorization  to  buy 
them. 

Sam  -  Systems  Analyst 

Sam  is  a  systems  analyst  in  the  Missile  Defense  Agency.  He  supports  the  COCOMs  by 
researching  items  on  their  integrated  priority  lists,  in  particular  looking  for  organizations  that  can 
support  specific  needs.  As  part  of  his  job  he  also  manages  a  science  and  technology  (S&T) 
community  page  with  information  about  upcoming  conferences,  points  of  contact,  a  blog  and  a 
discussion  area.  He  spends  “a  good  chunk”  of  each  workday  searching  for  information  for  his 
job.  Most  of  the  time,  he  uses  the  web.  He  monitors  DoDTechipedia  and  uses  other  DTIC 
products  a  few  times  per  week. 

Specific  Need  -  Sam  needs  to  find  an  official  organization  chart  for  part  of  the  MDA.  He  starts 
with  the  MDA  portal,  entering  a  search  for  “organization  chart.”  He  revises  his  query  several 
times,  uses  Boolean  combinations  of  AND  and  NOT,  and  examines  the  results.  After  two  hours 
he  finds  an  unofficial  version,  but  not  the  official  one.  He  is  frustrated  because  he  is  unable  to 
narrow  his  search  by  file  format  which  would  give  him  the  executed  (approved)  version. 

Exploratory  Need  -  Sam  needs  to  know  what  is  not  working  well  for  the  deployable  force 
protection  teams  and  what  kinds  of  S&T  solutions  are  available,  either  inside  or  outside  the  DoD. 
He  starts  at  the  point  of  what  is  known  about  a  given  problem  and  searches  using 
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DoDTechipedia  first.  He  then  searches  on  DTIC’s  unclassified  and  classified  systems.  Next  he 
takes  an  “outside”  perspective,  using  Google  to  see  what  parallel  information  is  available  from 
the  civilian  or  commercial  content  and  knowledge  environment.  He  also  monitors  specific 
known  web  sites.  For  example,  a  main  source  of  information  regarding  incidents  for  this  team  is 
JIEDO,  which  provides  after-action  reports  on  IED  incidents  with  the  force.  He  monitors  these 
reports,  as  well  as  other  internal  and  external  sources,  on  an  ongoing  basis.  He  shares  his 
findings  with  his  team  and  other  working  groups. 

Alan  -  USNR  officer 

Alan  is  an  expert  in  radioactive  materials,  which  involves  him  in  both  civilian  and  military 
working  environments.  He  is  retired  from  active  duty  in  the  US  Navy,  and  is  a  Captain  in  the  US 
Naval  Reserves.  He  is  a  radiation  safety  officer  for  a  research  university  and  is  a  senior  military 
representative  to  a  nearby  commercial  reactor.  He  is  also  a  technical  consultant  to  the  Joint 
Reserve  serving  the  Office  of  the  Secretary  of  Defense’s  Director  of  Defense  Research  & 
Engineering  (OSD  DDR&E)  activities,  a  unit  formed  four  years  ago  as  a  panel  of  subject  matter 
experts.  In  this  role,  he  serves  with  other  experts  to  provide  additional  knowledge,  experience 
and  training  to  DDR&E  in  technical  and  economic  aspects  by  brainstorming,  war  gaming  and 
researching.  He  uses  the  web  to  seek  information  daily.  In  his  role  as  technical  consultant  to 
DDR&E,  he  uses  DoDTechipedia  to  develop  content  pages  related  to  human  subjects  testing 
protocols.  He  works  on  this  task  2-3  times  per  week  and  says  his  skill  rate  on  Techipedia  is  still 
developing.  He  is  aware  of  other  DTIC  products,  but  doesn't  use  them  extensively. 

Specific  Need  -  Alan  recently  needed  to  verify  a  detail  about  the  proper  way  to  wear  his  uniform 
for  an  event.  He  frequently  needs  to  look  up  DoD  or  Department  of  the  Navy  instructions  or 
directives,  so  he  started  his  search  at  his  usual  starting  point  for  this  type  of  search,  the  Naval 
Personnel  Center  Reference  Library  (http://www.persnet.navv.mil/ReferenceLibrarv/) .  There  he 
finds  links  to  Directives  and  Instructions  and  from  there  to  uniform  regulations.  This  takes  him 
less  than  a  minute  to  locate.  The  first  time  he  needed  this  type  of  information  he  spent  3-4 
minutes  to  find  it  via  Google. 

Exploratory  Need  -  Alan  tracks  information  on  an  ongoing  basis  for  the  team  of  people  who 
discuss  energy  commodities  -  commodities  that  are  critical  for  the  functionality  of  the  US 
military.  In  particular,  he  monitors  topics  on  low  tech  energy  efficiency  as  well  as  on  R&D 
spending  on  new  or  renewable  energy  technologies.  He  looks  for  information  about  what 
production  changes  are  planned  for  these  sources.  He  begins  his  search  with  a  Google  query  for 
keywords  of  interest  so  that  he  can  learn  or  confirm  commodity  production  schedules  to  estimate 
impact  on  supply  chains.  He  searches  for  news  reports,  press  releases,  and  other  items  that  signal 
manufacturing  production  changes.  For  one  recent  round  of  research,  he  initially  spent  4  hours, 
and  did  not  find  the  answers  to  his  questions.  Although  he  has  expertise  in  the  production  and 
use  of  these  resources,  he  is  looking  for  information  on  company  business  plans  and  strategies. 
Due  to  budget  limitations  within  his  organization,  he  does  not  have  direct  access  to  subscription 
services  that  could  provide  analysis  regarding  that  kind  information  on  companies.  He 
participates  in  listserves  as  one  way  to  gather  information. 
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4.3  Other  Observations  and  Notes 


Several  interviewees  noted  that  there  is  a  big  difference  between  the  classified,  controlled  and 
public  environments  and  this  causes  a  large  gap  in  access  to  the  information  needed. 

There  are  policies  that  limit  access  to  materials  and  content  on  DoDTechipedia  -  restrictions  as 
to  who  can  get  on  to  that  site  will  impact  dissemination  of  this  content.  This  content  has  been 
pulled  in  from  various  other  sites  within  DoDTechipedia  and  external  to  DoDTechipedia  but 
access  is  limited  just  to  the  federal  government  and  contractors.  Users  are  still  figuring  out  how 
to  effectively  use  the  tool  -  what  information  can  and  should  be  posted  in  DoDTechipedia. 

Cost  and  context  of  the  need  for  the  information  effects  how  and  where  to  search  and  to  some 
degree  the  success  in  locating  the  material.  Several  interviewees  commented  on  not  having  funds 
to  purchase  reports  or  other  materials  that  were  relevant  to  their  search.  Instead,  they  chose  to 
keep  looking  for  freely  available  equivalent  or  acceptable  alternative  information  sources. 

One  contractor  stressed  that  she  is  very  mindful  of  segregating  her  sources  depending  upon  who 
the  search  is  for:  she  uses  appropriate  client -based  tools  or  public  sources  when  she  searches  for 
client  related  information,  and  utilizes  her  employer’s  information  tools  or  public  information 
when  working  on  market  research  analysis  projects. 

Other  tools/sources  referenced  during  the  interviews: 

•  DSTKOL.com 

•  npc.navy.mil/  -  reference  library 

•  blog:  Critical  Commodities  -  http://criticalcommodities.com/ 

•  ITIL  -  Information  Technology  Infrastructure  Library  -  (ITIL)  A  method  of  organizing 
the  system  and  network  management  departments  of  large  organizations.  ITIL  defines  the 
(work)  processes  involved  and  the  interfaces  between  them. 

•  Air  Force  Portal 

•  DoD  Document  search 

•  DoD  Statistics  Office 

•  Intellipedia 

•  DTIC  Research  and  Development  Descriptive  Summaries  (RDDS)  R&D  exhibits 

•  GAO  Reports  -  budget  exhibits 

•  Internal  senior  level  briefings 

•  Newton  -  beta  tool  for  project  communication  and  management? 

4.4  Discussion 

The  personas  and  scenarios  highlight  important  aspects  of  information  seeking  behavior  for  these 
end-users. 
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Google  is  the  single  most  common  starting  point  for  these  searchers,  but  they  consult  many  other 
sources,  applying  a  variety  of  strategies  and  tactics.  Some  of  their  sources  are  very  specific; 
others  are  more  general.  Searchers  have  a  plethora  of  sources  of  authoritative  (or  authoritative 
enough)  information.  The  sheer  abundance  of  information  may  make  it  impractical  for  searchers 
to  select  one  or  two  best  starting  points  for  a  search.  Indeed,  even  with  bookmarks  and  other 
devices,  users  may  not  remember  which  known  sources  to  consult  for  an  information  need.  In 
these  cases,  using  a  search  engine  like  Google  isn’t  simply  because  it’s  easy  -  it  may  be  the  most 
efficient  way  to  reach  the  desired  information. 

At  least  one  potential  user  of  DTIC  systems  finds  the  current  systems  cumbersome,  and  several 
other  comments  hint  at  this.  These  comments  probably  reflect  changing  user  expectations  as 
much  as  design  limitations.  They  highlight  the  importance  that  many  end-users  place  on 
streamlined  designs  that  get  them  good,  but  not  necessarily  the  best,  information.  In  an  era  when 
information  overload  is  more  of  a  problem  than  information  scarcity,  this  may  be  a  rational 
strategy  to  minimize  the  amount  of  time  and  effort  spent  searching. 

These  end-user  searchers  are  part  of  a  social  and  collaborative  environment.  They  are  not  all 
librarians  or  information  specialists,  but  they  share  what  they  learn  with  their  colleagues. 
Similarly,  they  draw  on  their  colleagues  for  information  and  suggestions  on  where  to  get  needed 
information.  This  holds  even  for  the  non-DoDTechipedia  users. 

4.5  Application  and  Limitations 

These  personas  and  scenarios  are  specific  examples  of  information  needs  drawn  from  real  end- 
users.  These  examples  may  be  useful  for  informing  DTIC’s  planning  and  service  design  process. 
In  particular,  they  may  help  DTIC  to  identify  unmet  information  needs  and  envision  new  tools 
and  services.  The  personas  and  scenarios  may  also  provide  useful  starting  points  for  usability 
tests. 

The  personas  and  scenarios  are  not,  however,  comprehensive  in  any  sense,  nor  do  they  fully 
represent  the  variety  of  DTIC  end-users  and  their  information  needs.  These  techniques  can  be 
used  in  conjunction  with  DTIC’s  established  system  development  processes.  The  in-depth 
interviews  allow  time  to  explore  information  needs  as  well  as  the  context  of  the  needs  -  an 
outcome  that  is  hard  to  achieve  in  a  survey.  Focusing  on  one  or  two  critical  incidents  helps 
minimize  generalizations  and  rationalizations.  Instead,  this  can  yield  specific  details  that  would 
be  overlooked  in  a  focus  group  or  in  short  format  discussions.  Using  personas  instead  of  specific 
people  can  help  reduce  certain  interviewee  concerns  by  making  it  clear  that  they  are  not  being 
asked  to  speak  in  an  official  capacity. 

5  Recommendations 

The  six  end-user  interviews  yielded  a  detailed  set  of  information  needs  for  a  narrow  slice  of 
DTIC’s  end-user  community  -  its  “market.”  I  recommend  that  DTIC  continue  to  develop  its 
understanding  of  non-traditional  end-users  and  their  needs.  It  can  do  this  by  first  developing  a 
market  segmentation  plan  that  articulates  the  type  of  users  that  DTIC  serves  within  its  large  and 
diverse  science,  technology,  research  and  development,  acquisition,  and  warfighter  communities. 
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Next,  DTIC  can  use  the  methodology  from  part  2  of  this  project  to  directly  contact  end-users  and 
develop  personas  and  user  scenarios.  In  particular,  it  may  be  helpful  to  use  a  “snowball”  strategy 
to  identify  end-user  informants  -  that  is,  ask  current  contacts  to  suggest  additional  contacts,  and 
then  repeat  as  needed. 

I  recommend  expanding  the  target  audience  to  interview  a  total  of  20-30  people  (including  the 
initial  six).  It  may  be  useful  to  consider  additional  user  profile  characteristics  and  “market 
segments”  as  a  way  to  identify  potential  interviewees.  Analyzing  the  data  from  a  broader  set  of 
interviews  is  likely  to  yield  additional  information  needs  and  contexts.  These  rich  descriptions 
can  be  used  to  stimulate  discussion  of  user  needs  within  DTIC.  Subsequently,  a  larger-scale 
survey  may  be  advisable  to  validate  the  interview  findings,  particularly  where  the  interviews 
suggest  the  potential  value  of  new  services. 

I  also  recommend  that  DTIC  consider  adding  end-users  to  advisory  committees  and  teams  like 
the  Customer  Interface  Task  Force  where  feasible.  This  will  help  more  directly  involve  and 
represent  the  needs  of  end-users.  It  may  be  useful  to  establish  an  ongoing  recruitment  effort  and 
have  end-users  serve  for  a  one- year  term,  to  gain  continuing  access  to  fresh  perspectives,  needs 
and  ideas. 

As  a  method  for  gathering  insight  into  technology  developments  not  yet  being  discussed  in  the 
trade  press,  DTIC  should  consider  hosting  a  workshop  with  experts  in  the  research  community. 
This  could  particularly  focus  on  changing  demographics  of  DTIC’ s  end-users  and  potential  end- 
users,  the  evolving  technology  and  what  needs  and  opportunities  these  trends  presents  for  DTIC. 
DTIC  should  continue  to  monitor  the  LIS  literature,  especially  conferences  and  journals  in  the 
area  of  human  computer  interaction  and  information  retrieval  (HCIR).  Researchers  are  using 
these  to  report  and  discuss  new  systems,  tools,  techniques  and  issues.  Tomorrow’s  enterprise- 
scale  search  interfaces  are  being  envisioned,  designed  and  evaluated  in  these  venues. 

Finally,  DTIC  should  continue  the  practice  of  environmental  scans,  including  research  journals 
related  to  information  needs  and  behaviors,  especially  of  the  “Millennial”  and  succeeding 
generations.  These  would  include  journals,  conferences  and  workshop  in  the  LIS  field,  as  well  as 
education,  sociology,  human-computer  interaction  (HCI),  psychology,  and  other  fields. 

6  Conclusion 

This  project  has  critiqued  elements  of  DTIC’ s  current  requirements  gathering  practices, 
suggesting  some  modest  refinements.  As  noted  above,  this  critique  was  narrowly  focused  on 
aspects  related  to  user  needs,  so  this  report  should  not  be  considered  comprehensive  of  the 
broader  requirements  analysis  or  design  process.  A  literature  review  was  used  to  extend  our 
understanding  of  user  needs  identification  to  DTIC's  domain:  defense-related  information 
seeking,  and  several  techniques  were  recommended  for  use  in  an  exploratory  study.  The 
exploratory  study  demonstrated  the  efficacy  of  these  techniques,  yielding  personas  and  user 
scenarios  that  illustrate  important  user  characteristics  and  needs  and  contributing  to  DTIC’s 
understanding  of  the  current  and  future  user  needs  of  its  user  communities. 
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Appendix  A  -  Documents  Reviewed 

The  following  documents  were  reviewed  during  part  1 : 

.  DTIC  Strategic  Plan  2007-2012 

•  Mission  and  Functions,  September  2007 

•  2008  Customer  Satisfaction  Survey 

•  Technical  report  "Current  Searching  Methodology  and  Retrieval  Issues:  An  Assessment" 
(March  2008) 

•  Documents  from  FAST  executive  briefing,  including  "The  1 1  Possible  Enhancements  in 
order  of  priority"  (July  15,  2009) 

•  Consolidated  Search  Requirements  (requirementsconsolodated090512.xls) 

•  DTIC  Online  web  site  (www.dtic.mil) 
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Appendix  B  -  Part  2  Activities  Description 


The  findings  of  Part  1  of  this  investigation  suggest  that  DTIC  has  effective  mechanisms  for  identifying 
the  needs  of  its  expert  users:  the  librarians  and  information  professionals  who  are  often  intermediaries 
working  on  behalf  of  the  end-user.  The  needs  of  informal  searchers  are  not  as  well  understood  as  the 
needs  of  the  professionals.  It  is  not  clear  who  comprises  the  informal  searchers.  DTIC’s  User  Services 
Directorate  indicates  that,  "We  usually  advertise  that  we  target  the  science,  technology,  research  and 
development,  acquisition,  and  warfighter  communities.  Our  Web  page  refers  to  serving  the  warfighter, 
researchers,  scientists,  engineers,  laboratories,  and  universities"  (private  communication).  However,  we 
have  found  no  document  yet  that  articulates  the  types  of  users  DTIC  seeks  to  serve  within  each  of  these 
large  and  diverse  communities.  This  is  an  important  step  in  a  deeper  understanding  of  the  needs  of  users. 
Articulating  specific  user  profiles  will  allow  DTIC  to  explore  the  information  needs  of  these  users  and 
develop  rich  usage  scenarios  that  complement  DTIC’s  rich  understanding  of  its  more  traditional  users  - 
experts  and  intermediaries. 

This  proposal  describes  two  activities  that  apply  techniques  reviewed  in  the  Part  1  report.  These  activities 
are  proposed  because  they  a)  will  provide  useful  information  for  DTIC  about  end-user  needs,  b)  they  are 
feasible  within  the  budget  and  schedule  constraints,  and  c)  they  can  (I  believe)  be  accomplished  without 
major  logistical  challenges  (e.g.,  a  survey  would  require  OPM  approval). 

The  two  activities  are: 

1 .  Develop  user  profiles  and  corresponding  personas  for  non-traditional  users  -  by  classifying  users 
according  to  relevant  information  needs,  work  contexts,  and  other  characteristics 

2.  Develop  usage  scenarios  based  on  the  developed  user  profiles  by  interviewing  non-traditional 
end-users 

The  focus  of  the  two  activities  will  be  non-traditional  users,  informal  searchers  who  are  not  necessarily 
experts  or  intermediaries. 

Activity  1:  User  Profiles  and  Personas 

Objective:  Develop  3-5  profiles  of  non-traditional  users  and  corresponding  personas 

Approach:  Identify  an  initial  set  of  user  profiles  using  the  existing  (limited)  literature  and  information 
from  Part  1.  Review  it  with  selected  DTIC  staff  and  refine  it.  Identify  a  set  of  informants  (ideally  2-3  per 
profile)  who  are  representative  of  these  users.  If  real  end-users  are  not  available,  it  may  be  sufficient  to 
work  with  informants  who  are  very  familiar  with  the  end-users  and  their  work.  Arrange  in-person  or 
telephone  interviews  (approximately  45  minutes  per  informant)  using  a  semi -structured  interview  to 
refine  and  elaborate  the  user  profiles.  Use  the  profiles  to  craft  personas,  and  then  review  the  profiles  and 
personas  with  informants  and  refine  as  needed. 

Activity  2:  Usage  Scenarios 

Objective:  Develop  1-3  usage  scenarios  for  each  user  profile  identified  in  activity  1. 
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Approach:  As  part  of  the  end-user  interviews  in  activity  1,  collect  in-depth  data  about  information  needs, 
work  contexts,  search  tasks,  and  other  related  activities.  Compile  this  data  into  sample  scenarios.  Email  to 
respondents  (with  the  profiles  and  personas)  and  follow-up  with  email  or  phone  call  to  collect  feedback 
on  the  scenarios,  and  refine  as  needed. 

Outcomes 

The  user  profiles,  personas  and  usage  scenarios  will  help  document  the  non-traditional  users  of  DTIC 
resources.  Although  this  information  will  be  based  on  a  small  sample  of  users,  it  will  provide  a  deeper 
description  of  these  users  which  could  be  complemented  with  a  broader  future  survey.  The  profiles, 
personas  and  usage  scenarios  will  be  useful  for  ongoing  usability  tests,  as  well  as  for  future  design  efforts. 
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Appendix  C  -  Part  2  Interview  Questions 


Semi-structured  Interview  Questions 

Introduction:  “Thank  you  for  agreeing  to  meet  with  me.  I  am  working  with  Carol  Jacobson  to  explore  the 
science  and  technical  information  needs  of  people  who  don’t  traditionally  work  directly  with  DTIC.  DTIC 
targets  the  science,  technology,  research  and  development,  acquisition,  and  warfighter  communities.  Your 
participation  will  help  us  to  better  understand  the  needs  of  specific  users  within  these  diverse 
communities.  I  have  some  questions  that  will  get  us  started,  but  we  aren’t  limited  by  them.  We’ll  see 
where  the  conversation  goes.  Before  we  get  started,  do  you  have  any  questions?  May  I  record  this 
interview?” 

Basic  information 

Job  title/rank 

Organization  -  employer  &  posting  agency  (if  contractor) 

Gender 

Educational  level  (HS,  Bachelors,  Masters,  PhD,  other  certifications) 

Age  (by  decade) 

Years  in  the  field  (by  decade) 


Web  information  seeking  frequency  -  How  often  do  you  use  the  Web  (e.g.  a  web  search  engine, 
browsing)  to  look  for  job-related  information?  [categorize  responses  according  to  following:] 
Less  than  1/month 

At  least  1/month,  but  less  than  1/week 
1  -2  times  per  week,  but  less  than  once  a  day 
At  least  once  a  day 


How  do  you  rate  your  Web  information  seeking  skills? 

Novice  1  2  3  4  5  Expert 


What  DTIC  products  do  you  use,  e.g.  DTIC  Online,  DTIC  Online  -  Access  Controlled,  and 
DoDTechipedia. 


General  DTIC  usage  frequency  -  How  often  do  you  use  a  DTIC  product  to  look  for  information? 
[may  get  multiple  product  responses] 

Less  than  1/month 

At  least  1/month,  but  less  than  1/week 
1-2  times  per  week,  but  less  than  once  a  day 
At  least  once  a  day 


How  do  you  rate  your  skill  with  the  DTIC  products  you  use? 

Novice  1  2  3  4  5  Expert 


Can  you  describe  your  organization  and  your  job  responsibilities?  (interested  in  how  people 
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classify  themselves  (scientists,  engineers,  information  professionals,  etc.)) 


I’d  like  to  ask  you  about  two  general  kinds  of  information  needs  you  may  have  had.  First,  what  is 
often  called  a  known  item  or  fact-finding  need.  This  is  where  you  need  to  find  a  specific 
document  or  fact,  or  get  a  specific  answer  to  a  question.  Second,  is  what  we  call  an  exploratory 
information  need.  These  often  are  more  complex,  you  might  need  to  find  multiple  documents,  and 
you  might  need  to  look  for  information  in  multiple  places  or  conduct  multiple  searches.  You 
might  not  even  be  quite  sure  of  what  you  are  looking  for  -  it  might  be  a  bit  ambiguous  or 
uncertain.  Of  course,  real  information  needs  aren’t  cut  and  dried,  so  your  needs  might  fall 
somewhere  in  between  these  extremes.  We  are  especially  interested  if  you  used  DoDTechipedia, 
Online  -  Access  Controlled  or  other  DTIC  products.  Can  you  start  by  telling  me  about  a  known 
item  or  fact-finding  need?  [see  sub-questions  below]  Now  can  you  tell  me  about  a  more  complex 
or  exploratory  need? 

[For  each  need,  explore:] 

Goal,  source  of  task  -  [May  need  to  provide  guidance  to  appropriately  describe  high  level 
work  task  that  motivated  the  information  need.] 

Type  and  amount  of  information  needed 

Who  needed  the  information  (you  personally,  colleague,  boss,  etc. 

Time  aspects  -  frequency  (one-time  vs.  repeated),  length,  stage,  time  constraints,  urgency 
Importance  (salience) 

Complexity  (e.g.  Bystrom  &  Jarvelin) 

Difficulty 

Can  you  characterize  your  prior  knowledge  of  the  domain 
What  resources  were  consulted  (human  too)? 

How  did  you  ultimately  satisfy  the  need? 

Did  you  work  with  others  on  this  information  need?  In  what  ways  did  you  collaborate? 
How  might  this  task  differ  from  non-military  info  needs? 

Effect  of  any  security  and  confidentiality  issues? 

Effect  of  other  organizational  policies? 


Can  you  tell  me  about  your  DoDTechipedia  page?  When  started?  How  many  people  write/read? 
What  kind  of  research  did  you  do  to  put  together? 


May  we  contact  you  with  follow  up  questions? 


Can  you  suggest  any  colleagues  who  we  could  contact? 
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